CUBRID HA¶

Nodes and Groups¶

A node is a logical unit that makes up CUBRID HA. It can become one of the following nodes according to its status: master node, slave node, or replica node.

• Master node : A node to be replicated. It provides all services which are read, write, etc. using an active server.
• Slave node : A node that has the same information as a master node. Changes made in the master node are automatically reflected to the slave node. It provides the read service using a standby server, and a failover will occur when the master node fails.
• Replica node : A node that has the same information as a master node. Changes made in the master node are automatically reflected to the replica node. It provides the read service using a standby server, and no failover will occur when the master node fails.

The CUBRID HA group consists of the nodes described above. You can configure the members of this group by using the ha_node_list and ha_replica_list in the cubrid_ha.conf file. Nodes in a group have the same information. They exchange status checking messages periodically and a failover will occurs when the master node fails.

A node includes the master process (cub_master), the database server process (cub_server), the replication log copy process (copylogdb), the replication log reflection process (applylogdb), etc.

Processes¶

A CUBRID HA node consists of one master process (cub_master), one or more database server processes (cub_server), one or more replication log copy processes (copylogdb), and one or more replication log reflection processes (applylogdb). When a database is configured, database server processes, replication log copy processes, and replication log reflection processes will start. Because copy and reflection of a replication log are executed by different processes, the delay in replicating reflections does not affect the transaction that is being executed.

• Master process (cub_master) : Exchanges heartbeat messages to control the internal management processes of CUBRID HA.
• Database server process (cub_server) : Provides services such as read or write to the user. For details, see Servers.
• Replication log copy process (copylogdb) : Copies all transaction logs in a group. When the replication log copy process requests a transaction log from the database server process of the target node, the database server process returns the corresponding log. The location of copied transaction logs can be configured in the ha_copy_log_base of cubrid_ha.conf. Use applyinfo utility to verify the information of copied replication logs. The replication log copy process has following two modes: SYNC and ASYNC. You can configure it with the ha_copy_sync_mode of cubrid_ha.conf. For details on these modes, see Log Multiplexing.

• Replication log reflection process (applylogdb) : Reflects the log that has been copied by the replication log copy process to a node. The information of reflected replications is stored in the internal catalog (db_ha_apply_info). You can use the applyinfo utility to verify this information.

Servers¶

Here, the word "server" is a logical representation of database server processes. Depending on its status, a server can be either an active server or a standby server.

• Active server : A server that belongs to a master node; the status is active. An active server provides all services, including read, write, etc. to the user.
• Standby server : A standby server that belongs to a non-master node; the status is standby. A standby server provides only the read service to the user.

The server status changes based on the status of the node. You can use the cubrid changemode utility to verify server status. The maintenance mode exists for operational convenience and you can change it by using the cubrid changemode utility.

• active : The status of servers that run on a master node is usually active. In this status, all services including read, write, etc. are provided.
• standby : The status of servers that run on a slave node or a replica node is standby. In this status, only the read service is provided.
• maintenance : The status of servers can be manually changed for operational convenience is maintenance. In this status, only a csql can access and no service is provided to the user.
• to-be-active : The status in which a standby server will become active for reasons such as failover, etc. is to-be-active. In this status, servers prepare to become active by reflecting transaction logs from the existing master node to its own server. The node in this status can accept only SELECT query.
• Other : This status is internally used.

When the node status is changed, on cub_master process log and cub_server process log, following error messages are saved. But, they are saved only when the value of error_log_level in cubrid.conf is error or less.

• The following log information of cub_master process is saved on $CUBRID/log/<hostname>_master.err file.

  HA generic: Send changemode request to the server. (state:1[active], args:[cub_server demodb ], pid:25728).
  HA generic: Receive changemode response from the server. (state:1[active], args:[cub_server demodb ], pid:25728).
  

• The following log information of cub_server is saved on $CUBRID/log/server/<db_name>_<date>_<time>.err file.

  Server HA mode is changed from 'to-be-active' to 'active'.
  

heartbeat Message¶

As a core element to provide HA, it is a message exchanged among master, slave, and replica nodes to monitor the status of other nodes. A master process periodically exchanges heartbeat messages with all other master processes in the group. A heartbeat message is exchanged through the UDP port configured in the ha_port_id parameter of cubrid_ha.conf. The exchange interval of heartbeat messages is determined by an internally configured value.

When the master node fails, a failover occurs to a slave node.

failover and failback¶

A failover means that the highest priority slave node automatically becomes a new master node when the original master node fails to provide services due to a failure. A master process calculates scores for all nodes in the CUBRID HA group based on the collected information, promotes slave nodes to master modes when it is necessary, and then notifies the management process of the changes it has made.

A failback means that the previously failed master node automatically becomes a master node back after the failure node is restored. The CUBRID HA does not currently support this functionality.

If a heartbeat message fails to deliver, a failover will occur. For this reason, servers with unstable connection may experience failover even though no actual failures occur. To prevent a failover from occurring in the situation described above, configure ha_ping_ports. Configuring ha_ping_ports will send a ping message to a node specified in ha_ping_ports in order to verify whether the network is stable or not when a heartbeat message fails to deliver. For details on configuring ha_ping_ports, see cubrid_ha.conf.

Broker Mode¶

A broker can access a server with one of the following modes: Read Write, Read Only or Standby Only. This configuration value is determined by a user.

A broker finds and connects to a suitable DB server by trying to establish a connection in the order of DB server connections; this is, if it fails to establish a connection, it tries another connection to the next DB server defined until it reaches the last DB server. If no connection is made even after trying all servers, the broker fails to connect to a DB server.

For details on how to configure broker mode, see cubrid_broker.conf.

DB connection is affected by PREFERRED_HOSTS, CONNECT_ORDER and MAX_NUM_DELAYED_HOSTS_LOOKUP parameters in cubrid_broker.conf. See Connecting a Broker to DB for further information.

The below is the description if the above parameters are not specified.

Read Write

"ACCESS_MODE=RW"

A broker that provides read and write services. This broker is usually connected to an active server. If there is no active server, this broker will be connected to a standby server temporarily. Therefore, a Read Write broker can be temporarily connected to a standby server.

When the broker temporarily establishes a connection to a standby server, it will disconnect itself from the standby server at the end of every transaction so that it can attempt to find an active server at the beginning of the next transaction. When it is connected to the standby server, only read service is available. Any write requests will result in a server error.

The following picture shows how a broker connects to the host through the db-host configuration.

The broker tries to connect as the order of B, C, A because db-host in databases.txt is "node B:node C:node A". At this time, "node B:node C:node A" specified in db-host is the real host names defined in the /etc/hosts file.

• Example 1. node B is crashed, node C is in standby status, and node A is in active status. Therefore, at last, the broker connects to node A.
• Example 2. node B is crashed, and node C is in active status. Therefore, at last, the broker connects to node C.

Read Only

"ACCESS_MODE=RO"

A broker that provides the read service. This broker is connected to a standby server if possible. Therefore, the Read Only broker can be connected to an active server temporarily.

Once it establishes a connection with an active server, it will maintain that connection until the time specified by RECONNECT_TIME. After RECONNECT_TIME, the broker tries to reconnect as disconnecting the old connection. Or you can reconnect to the standby server by running cubrid broker reset. If a write request is delivered to the Read Only broker, an error occurs in the broker; therefore, only the read service will be available even if it is connected to an active server.

The following picture shows how a broker connects to the host through the db-host configuration.

The broker tries to connect as the order of A, B, C because db-host in databases.txt is "node A:node B:node C". At this time, "node A:node B:node C" specified in db-host is the real host names defined in the /etc/hosts file.

• Example 1. node A is in active status, node B is in standby status. Therefore, at last, the broker connects to node B.
• Example 2. node A is in active status, node B is crashed, and node C is in standby status. Therefore, at last, the broker connects to node C.
• Example 3. node A is in active status, node B and node C are crashed. Therefore, at last, the broker connects to node A.

Standby Only

"ACCESS_MODE=SO"

A broker that provides the read service. This broker can only be connected to a standby server. If no standby server exists, no service will be provided.

The following picture shows how a broker connects to the host through the db-host configuration.

The broker tries to connect as the order of A, B, C because db-host in databases.txt is "node A:node B:node C". At this time, "node A:node B:node C" specified in db-host is the real host names defined in the /etc/hosts file.

• Example 1. node A is in active status, node B is in standby status. Therefore, at last, the broker connects to node B.
• Example 2. node A is in active status, node B is crashed, and node C is in standby status. Therefore, at last, the broker connects to node C.
• Example 3. node A is in active status, node B and node C are crashed. Therefore, at last, the broker does not connect to any node. This is the difference with Read Only broker.

Duplexing Servers¶

Duplexing servers is building a system by configuring duplicate hardware equipment to provide CUBRID HA. This method will prevent any interruptions in a server in case of occurring a hardware failure.

Server failover

A broker defines server connection order and connects to a server according to the defined order. If the connected server fails, the broker connects to the server with the next highest priority. This requires no processing in the application side. The actions taken when the broker connects to another server may differ according to the current mode of the broker. For details on the server connection order and configuring broker mode, see cubrid_broker.conf.

Server failback

CUBRID HA does not automatically support server failback. Therefore, to manually apply failback, restore the master node that has been abnormally terminated and run it as a slave node, terminate the node that has become the master from the slave due to failover, and finally, change the role of each node again.

For example, when nodeA is the master and nodeB is the slave, nodeB becomes the master and nodeA becomes the slave after a failover. After terminating nodeB (cubrid heartbeat stop) check (cubrid heartbeat status) whether the status of nodeA has become active. Start (cubrid heartbeat start) nodeB and it will become the slave.

Duplexing Brokers¶

As a 3-tier DBMS, CUBRID has middleware called the broker which relays applications and database servers. To provide HA, the broker also requires duplicate hardware equipment. This method will prevent any interruptions in a broker in case of occurring a hardware failure.

The configuration of broker redundancy is not determined by the configuration of server redundancy; it can be user-defined. In addition, it can be separated by piece of individual equipment.

To use the failover and failback functionalities of a broker, the altHosts attribute must be added to the connection URL of the JDBC, CCI, or PHP. For a description of this, see JDBC Configuration, CCI Configuration and PHP Configuration.

To set a broker, configure the cubrid_broker.conf file. To set the order of failovers of a database server, configure the databases.txt file. For more information, see Configuring and Starting Broker, and Verifying the Broker Status.

The following is an example in which two Read Write (RW) brokers are configured. When the first connection broker of the application URL is set to broker B1 and the second connection broker to broker B2, the application connects to broker B2 when it cannot connect to broker B1. When broker B1 becomes available again, the application reconnects to broker B1.

The following is an example in which the Read Write (RW) broker and the Read Only (RO) broker are configured in each piece of equipment of the master node and the slave node. First, the app1 and the app2 URL connect to broker A1 (RW) and broker B2 (RO), respectively. The second connection (altHosts) is made to broker A2 (RO) and broker B1 (RW). When equipment that includes nodeA fails, app1 and the app2 connect to the broker that includes nodeB.

The following is an example of a configuration in which broker equipment includes one Read Write broker (master node) and two Preferred Host Read Only brokers (slave nodes). The Preferred Host Read Only brokers are connected to nodeB and nodeC to distribute the reading load.

Broker failover

The broker failover is not automatically failed over by the settings of system parameters. It is available in the JDBC, CCI, and PHP applications only when broker hosts are configured in the altHosts of the connection URL. Applications connect to the broker with the highest priority. When the connected broker fails, the application connects to the broker with the next highest priority. Configuring the altHosts of the connection URL is the only necessary action, and it is processed in the JDBC, CCI, and PHP drivers.

Broker failback

If the failed broker is recovered after a failover, the connection to the existing broker is terminated and a new connection is established with the recovered broker which has the highest priority. This requires no processing in the application side as it is processed within the JDBC, CCI, and PHP drivers. Execution time of failback depends on the value configured in JDBC connection URL. For details, see JDBC Configuration.

Log Multiplexing¶

CUBRID HA keeps every node in the CUBRID HA group with the identical structure by copying and reflecting transaction logs to all nodes included in the CUBRID HA group. As the log copy structure of CUBRID HA is a mutual copy between the master and the slave nodes, it has a disadvantage of increasing the size of a log volume. However, it has an advantage of flexibility in terms of configuration and failure handling, comparing to the chain-type copy structure.

The transaction log copy modes include SYNC and ASYNC. This value can be configured by the user in cubrid_ha.conf file.

SYNC Mode

When transactions are committed, the created transaction logs are copied to the slave node and stored as a file. The transaction commit is complete after receiving a notice on its success. Although the time to execute commit in this mode may take longer than that in ASYNC mode, this is the safest method because the copied transaction logs are always guaranteed to be reflected to the standby server even if a failover occurs.

ASYNC Mode

When transactions are committed, commit is complete without verifying the transfer of transaction logs to a slave node. Therefore, it is not guaranteed that committed transactions are reflected to a slave node in a master node side.

Although ASYNC mode provides a better performance as it has almost no delay when executing commit, there may be data inconsistency in its nodes.

Preparation¶

Structure Diagram

The diagram below aims to help users who are new to CUBRID HA, by explaining a simple procedure of the CUBRID HA configuration.

Specifications

Linux and CUBRID version 2008 R2.2 or later must be installed on the equipment to be used as the master and the slave nodes. CUBRID HA does not support Windows operating system.

Specifications of Configuring the CUBRID HA Equipment

Creating Databases and Configuring Servers¶

Creating Databases

Create databases to be included in CUBRID HA at each node of the CUBRID HA in the same manner. Modify the options for database creation as needed.

[nodeA]$ cd $CUBRID_DATABASES
[nodeA]$ mkdir testdb
[nodeA]$ cd testdb
[nodeA]$ mkdir log
[nodeA]$ cubrid createdb -L ./log testdb en_US
Creating database with 512.0M size. The total amount of disk space needed is 1.5G.

CUBRID 10.0

[nodeA]$

cubrid.conf

Ensure ha_mode of $CUBRID/conf/cubrid.conf in every CUBRID HA node has the same value. Especially, take caution when configuring the log_max_archives and force_remove_log_archives parameters (logging parameters) and the ha_mode parameter (HA parameter).

# Service parameters
[service]
service=server,broker,manager

# Common section
[common]
service=server,broker,manager

# Server parameters
server=testdb
data_buffer_size=512M
log_buffer_size=4M
sort_buffer_size=2M
max_clients=100
cubrid_port_id=1523
db_volume_size=512M
log_volume_size=512M

# Adds when configuring HA (Logging parameters)
log_max_archives=100
force_remove_log_archives=no

# Adds when configuring HA (HA mode)
ha_mode=on

cubrid_ha.conf

Ensure ha_port_id, ha_node_list, ha_db_list of $CUBRID/conf/cubrid_ha.conf in every CUBRID HA node has the same value. In the example below, we assume that the host name of a master node is nodeA and that of a slave node is nodeB.

[common]
ha_port_id=59901
ha_node_list=cubrid@nodeA:nodeB
ha_db_list=testdb
ha_copy_sync_mode=sync:sync
ha_apply_max_mem_size=500

databases.txt

Ensure that you must configure the host names (nodeA:nodeB) of master and slave nodes in db-host of $CUBRID_DATABASES/databases.txt; if $CUBRID_DATABASES is not configured, do it in $CUBRID/databases/databases.txt).

#db-name vol-path db-host log-path lob-base-path
testdb /home/cubrid/DB/testdb nodeA:nodeB /home/cubrid/DB/testdb/log file:/home/cubrid/DB/testdb/lob

Starting and Verifying CUBRID HA¶

Starting CUBRID HA

Execute the cubrid heartbeat start at each node in the CUBRID HA group. Note that the node executing cubrid heartbeat start first will become a master node. In the example below, we assume that the host name of a master node is nodeA and that of a slave node is nodeB.

• Master node

  [nodeA]$ cubrid heartbeat start
  

• Slave node

  [nodeB]$ cubrid heartbeat start
  

Verifying CUBRID HA Status

Execute cubrid heartbeat status at each node in the CUBRID HA group to verify its configuration status.

[nodeA]$ cubrid heartbeat status
@ cubrid heartbeat list
 HA-Node Info (current nodeA-node-name, state master)
   Node nodeB-node-name (priority 2, state slave)
   Node nodeA-node-name (priority 1, state master)
 HA-Process Info (nodeA 9289, state nodeA)
   Applylogdb testdb@localhost:/home1/cubrid1/DB/testdb_nodeB.cub (pid 9423, state registered)
   Copylogdb testdb@nodeB-node-name:/home1/cubrid1/DB/testdb_nodeB.cub (pid 9418, state registered)
   Server testdb (pid 9306, state registered_and_active)

[nodeA]$

Use the cubrid changemode utility at each node in the CUBRID HA group to verify the status of the server.

• Master node

  [nodeA]$ cubrid changemode testdb@localhost
  The server 'testdb@localhost''s current HA running mode is active.
  

• Slave node

  [nodeB]$ cubrid changemode testdb@localhost
  The server 'testdb@localhost''s current HA running mode is standby.
  

Verifying the CUBRID HA Operation

Verify that action is properly applied to standby server of the slave node after performing write in an active server of the master node. To make a success connection via the CSQL Interpreter in HA environment, you must specify the host name to be connected after the database name like "@<host_name>"). If you specify a host name as localhost, it is connected to local node.

• Master node

  [nodeA]$ csql -u dba testdb@localhost -c "create table abc(a int, b int, c int, primary key(a));"
  [nodeA]$ csql -u dba testdb@localhost -c "insert into abc values (1,1,1);"
  [nodeA]$
  

• Slave node

  [nodeB]$ csql -u dba testdb@localhost -l -c "select * from abc;"
  === <Result of SELECT Command in Line 1> ===
  <00001> a: 1
          b: 1
          c: 1
  [nodeB]$
  

Configuring and Starting Broker, and Verifying the Broker Status¶

Configuring the Broker

To provide normal service during a database failover, it is necessary to configure an available database node in the db-host of databases.txt. And ACCESS_MODE in the cubrid_broker.conf file must be specified; if it is omitted, the default value is configured to Read Write mode. If you want to divide into a separate device, you must configure cubrid_broker.conf and databases.txt in the broker device.

• databases.txt

  #db-name        vol-path                db-host         log-path        lob-base-path
  testdb          /home1/cubrid1/CUBRID/testdb  nodeA:nodeB        /home1/cubrid1/CUBRID/testdb/log file:/home1/cubrid1/CUBRID/testdb/lob
  

• cubrid_broker.conf

  [%testdb_RWbroker]
  SERVICE                 =ON
  BROKER_PORT             =33000
  MIN_NUM_APPL_SERVER     =5
  MAX_NUM_APPL_SERVER     =40
  APPL_SERVER_SHM_ID      =33000
  LOG_DIR                 =log/broker/sql_log
  ERROR_LOG_DIR           =log/broker/error_log
  SQL_LOG                 =ON
  TIME_TO_KILL            =120
  SESSION_TIMEOUT         =300
  KEEP_CONNECTION         =AUTO
  CCI_DEFAULT_AUTOCOMMIT  =ON
  
  # broker mode parameter
  ACCESS_MODE             =RW
  

Starting Broker and Verifying its Status

A broker is used to access applications such as JDBC, CCI or PHP. Therefore, to simply test server redundancy, execute the CSQL interpreter that is directly connected to the server processes, without having to start a broker. To start a broker, execute cubrid broker start. To stop it, execute cubrid broker stop.

The following example shows how to execute a broker from the master node.

[nodeA]$ cubrid broker start
@ cubrid broker start
++ cubrid broker start: success
[nodeA]$ cubrid broker status
@ cubrid broker status
% testdb_RWbroker
---------------------------------------------------------
ID   PID   QPS   LQS PSIZE STATUS
---------------------------------------------------------
 1  9532     0     0  48120  IDLE

Configuring Applications

Specifies the host name (nodeA_broker, nodeB_broker) and port for an application to connect in the connection URL. The altHosts attribute defines the broker where the next connection will be made when the connection to a broker fails. The following is an example of a JDBC program. For more information on CCI and PHP, see CCI Configuration and PHP Configuration.

Connection connection = DriverManager.getConnection("jdbc:CUBRID:nodeA_broker:33000:testdb:::?charSet=utf-8&altHosts=nodeB_broker:33000", "dba", "");

cubrid.conf¶

The cubrid.conf file that has general information on configuring CUBRID is located in the $CUBRID/conf directory. This page provides information about cubrid.conf parameters used by CUBRID HA.

# Service Parameters
[service]
service=server,broker,manager

# Server Parameters
server=testdb
data_buffer_size=512M
log_buffer_size=4M
sort_buffer_size=2M
max_clients=200
cubrid_port_id=1523
db_volume_size=512M
log_volume_size=512M

# Adds when configuring HA (Logging parameters)
log_max_archives=100
force_remove_log_archives=no

# Adds when configuring HA (HA mode)
ha_mode=on
log_max_archives=100

cubrid_ha.conf¶

The cubrid_ha.conf file that has generation information on CUBRID HA is located in the $CUBRID/conf directory. CUBRID HA does not support Windows; it supports Linux only.

See Connecting a Broker to DB for further information regarding the process connecting between a broker and a DB server.

Time: 04/10/12 17:49:45.030 - ERROR *** file ../../src/connection/tcp.c, line 121 ERROR CODE = -353 Tran = 0, CLIENT = (unknown):(unknown)(-1), EID = 1 Cannot make connection to master server on host "Wrong_HOST_NAME".... Connection timed out

[common]
ha_node_list=cubrid@nodeA:nodeB
ha_db_list=testdb
ha_copy_sync_mode=sync:sync
ha_apply_max_mem_size=500

127.0.0.1 localhost.localdomain localhost
192.168.0.1 nodeA

cubrid_broker.conf¶

The cubrid_broker.conf file that has general information on configuring CUBRID broker is located in the $CUBRID/conf directory. This section explains the parameters of cubrid_broker.conf that are used by CUBRID HA.

See Connecting a Broker to DB for further information regarding the process connecting between a broker and a DB server.

[%PHRO_broker]
SERVICE                 =ON
BROKER_PORT             =33000
MIN_NUM_APPL_SERVER     =5
MAX_NUM_APPL_SERVER     =40
APPL_SERVER_SHM_ID      =33000
LOG_DIR                 =log/broker/sql_log
ERROR_LOG_DIR           =log/broker/error_log
SQL_LOG                 =ON
TIME_TO_KILL            =120
SESSION_TIMEOUT         =300
KEEP_CONNECTION         =AUTO
CCI_DEFAULT_AUTOCOMMIT  =ON

# Broker mode setting parameter
ACCESS_MODE             =RO
PREFERRED_HOSTS         =localhost

databases.txt¶

The databases.txt file has information on the order of servers for the CAS of a broker to connect. It is located in the $CUBRID_DATABASES (if not specified, $CUBRID/databases) directory; the information can be configured by using db_hosts. You can specify multiple nodes by using a colon (:). If "CONNECT_ORDER=RANDOM", the connection order is decided as randomly. But if PREFERRED_HOSTS is specified, the specified hosts have the first priority of the connection order.

The following example shows how to configure databases.txt.

#db-name    vol-path        db-host     log-path     lob-base-path
testdb       /home/cubrid/DB/testdb nodeA:nodeB   /home/cubrid/DB/testdb/log  file:/home/cubrid/DB/testdb/lob

JDBC Configuration¶

To use CUBRID HA in JDBC, you must specify the connection information of another broker (nodeB_broker) to be connected when a failure occurs in broker (nodeA_broker). The attribute configured for CUBRID HA is altHosts which represents information of one or more broker nodes to be connected. For details, see Configuration Connection.

The following example shows how to configure JDBC:

Connection connection = DriverManager.getConnection("jdbc:CUBRID:nodeA_broker:33000:testdb:::?charSet=utf-8&altHosts=nodeB_broker:33000", "dba", "");

CCI Configuration¶

To use CUBRID HA in CCI, you must use the cci_connect_with_url() function which additionally allows specifying connection information in connection URL; the connection information is used when a failure occurs in broker. The attribute configured for CUBRID HA is altHosts which represents information of one or more broker nodes to be connected.

The following example shows how to configure CCI.

con = cci_connect_with_url ("cci:CUBRID:nodeA_broker:33000:testdb:::?altHosts=nodeB_broker:33000", "dba", NULL);
if (con < 0)
{
      printf ("cannot connect to database\n");
      return 1;
}

PHP Configuration¶

To use the functions of CUBRID HA in PHP, connect to the broker by using cubrid_connect_with_url, which is used to specify the connection information of the failover broker in the connection URL. The attribute specified for CUBRID HA is altHosts, the information on one or more broker nodes to be connected when a failover occurs.

The following example shows how to configure PHP.

<?php
$con = cubrid_connect_with_url ("cci:CUBRID:nodeA_broker:33000:testdb:::?altHosts=nodeB_broker:33000", "dba", NULL);
if ($con < 0)
{
      printf ("cannot connect to database\n");
      return 1;
}
?>

Connection Process¶

When a broker accesses DB server, it tries the primary connection; if it fails, it tries the secondary connection.

• The primary connection: Check the DB status(active/standby) and the delay of replication.
  1. A broker tries to connect as the order specified by PREFERRED_HOSTS. The broker rejects connecting to the improper DB of which the status does not match with ACCESS_MODE or in which the replication is delayed.
  2. By the CONNECT_ORDER, a broker tries to connect to the host in the order specified in databases.txt or the random order. The broker checks the DB status followed by the ACCESS_MODE and checks the replication-delayed host up to the number specified in MAX_NUM_DELAYED_HOSTS_LOOKUP.
• The secondary connection: Ignore the DB status(active/standby) and the delay of replication. However, SO broker always accepts to connect only to standby DB.
  1. A broker tries to connect as the order specified by PREFERRED_HOSTS. The broker accepts connecting to the improper DB of which status does not match with ACCESS_MODE or in which the replication is delayed. However, SO broker can never connect to active DB.
  2. By the CONNECT_ORDER, a broker tries to connect to the host in the order specified in databases.txt or the random order. The broker ignores the DB status(active/standby) and the delay of replication; it is connected if possible.

Examples on Behaviours by Configuration¶

The following shows the example of configuration.

Host DB status

• host1: active
• host2: standby, replication is delayed.
• host3: standby, replica, unable to access.
• host4: standby, replica, replication is delayed.
• host5: standby, replica, replication is delayed.

When the status of host DBs are as the above, the below shows samples of behaviours by the configuration.

Behaviours by configuration

• 2-1, 2-2, 2-3: From 2, (+) is addition and (#) is modification.
• 3-1, 3-2, 3-3: From 3, (+) is addition and (#) is modification.

cubrid heartbeat Utility¶

cubrid heartbeat command can be run as cubrid hb, the abbreviated command.

$ cubrid heartbeat start

$ cubrid heartbeat start testdb

$ cubrid heartbeat stop

$ cubrid heartbeat stop testdb

$ cubrid heartbeat stop -i
or
$cubrid heartbeat stop --immediately

$ cubrid heartbeat copylogdb <start|stop> [ -h <host-name> ] db_name peer_node

<host-name>: the name of the remote host where copylogdb command will be executed

[nodeB]$ cubrid heartbeat copylogdb stop testdb nodeA
[nodeB]$ cubrid heartbeat copylogdb start testdb nodeA

$ cubrid heartbeat applylogdb <start|stop> [ -h <host-name> ] db_name peer_node

<host-name>: the name of the remote host where appplylogdb command will be executed

[nodeB]$ cubrid heartbeat applylogdb stop testdb nodeA
[nodeB]$ cubrid heartbeat applylogdb start testdb nodeA

$ cubrid heartbeat reload

cubrid heartbeat repl start <node_name>

cubrid heartbeat repl stop <node_name>

$ cubrid heartbeat status [-v] [ -h <host-name> ]

<host-name>: the name of the remote host where status command will be executed

$ cubrid heartbeat status
@ cubrid heartbeat status

 HA-Node Info (current nodeB, state slave)
   Node nodeB (priority 2, state slave)
   Node nodeA (priority 1, state master)


 HA-Process Info (master 2143, state slave)
   Applylogdb testdb@localhost:/home/cubrid/DB/testdb_nodeB (pid 2510, state registered)
   Copylogdb testdb@nodeA:/home/cubrid/DB/testdb_nodeA (pid 2505, state registered)
   Server testdb (pid 2393, state registered_and_standby)

Registering HA to cubrid service¶

If you register heartbeat to CUBRID service, you can use the utilities of cubrid service to start, stop or check all the related processes at once. The processes specified by service parameter in [service] section in cubrid.conf file are registered to CUBRID service. If this parameter includes heartbeat, you can start/stop all the service processes and the HA-related processes by using cubrid service start / stop command.

How to configure cubrid.conf file is shown below.

# cubrid.conf

...

[service]

...

service=broker,heartbeat

...

[common]

...

ha_mode=on

applyinfo¶

This utility is used to check the copied and applied status of replication logs by CUBRID HA.

cubrid applyinfo [option] <database-name>

• database-name : Specifies the name of a server to monitor. A node name is not included.

The following shows the [options] used on cubrid applyinfo.

-r, --remote-host-name=HOSTNAME¶

Configures the name of a target node in which transaction logs are copied. Using this option will output the information of active logs (Active Info.) of a target node.

-a, --applied-info¶

Outputs the information of replication reflection of a node executing cubrid applyinfo. The -L option is required to use this option.

-L, --copied-log-path=PATH¶

Configures the location of transaction logs copied from the other node. Using this option will output the information of transaction logs copied (Copied Active Info.) from the other node.

-p, --pageid=ID¶

Outputs the information of a specific page in the copied logs. This is available only when the -L option is enabled. The default is 0, it means the active page.

-v¶

Outputs detailed information.

-i, --interval=SECOND¶

Outputs the copied status and applied status of transaction logs per specified seconds. To see the delayed status of the replicated log, this option is mandatory.

Example

The following example shows how to check log information (Active Info.) of the master node, the status information of log copy (Copied Active Info.) of the slave node, and the applylogdb info (Applied Info.) of the slave node by executing applyinfo in the slave node.

• Applied Info.: Shows the status information after the slave node applies the replication log.
• Copied Active Info.: Shows the status information after the slave node copies the replication log.
• Active Info.: Shows the status information after the master node records the transaction log.
• Delay in Copying Active Log: Shows the status information which the transaction logs' copy is delayed.
• Delay in Applying Copied Log: Shows the status information which the transaction logs' application is delayed.

[nodeB] $ cubrid applyinfo -L /home/cubrid/DB/testdb_nodeA -r nodeA -a -i 3 testdb

 *** Applied Info. ***
Insert count                   : 289492
Update count                   : 71192
Delete count                   : 280312
Schema count                   : 20
Commit count                   : 124917
Fail count                     : 0

 *** Copied Active Info. ***
DB name                        : testdb
DB creation time               : 04:29:00.000 PM 11/04/2012 (1352014140)
EOF LSA                        : 27722 | 10088
Append LSA                     : 27722 | 10088
HA server state                : active

 *** Active Info. ***
DB name                        : testdb
DB creation time               : 04:29:00.000 PM 11/04/2012 (1352014140)
EOF LSA                        : 27726 | 2512
Append LSA                     : 27726 | 2512
HA server state                : active

 *** Delay in Copying Active Log ***
Delayed log page count         : 4
Estimated Delay                : 0 second(s)

 *** Delay in Applying Copied Log ***
Delayed log page count         : 1459
Estimated Delay                : 22 second(s)

The items shown by each status are as follows:

• Applied Info.
  • Committed page: The information of committed pageid and offset of a transaction reflected last through replication log reflection process. The difference between this value and the EOF LSA of "Copied Active Info. represents the amount of replication delay.
  • Insert Count: The number of Insert queries reflected through replication log reflection process.
  • Update Count: The number of Update queries reflected through replication log reflection process.
  • Delete Count: The number of Delete queries reflected through replication log reflection process.
  • Schema Count: The number of DDL statements reflected through replication log reflection process.
  • Commit Count: The number of transactions reflected through replication log reflection process.
  • Fail Count: The number of DML and DDL statements in which log reflection through replication log reflection process fails.
• Copied Active Info.
  • DB name: Name of a target database in which the replication log copy process copies logs
  • DB creation time: The creation time of a database copied through replication log copy process
  • EOF LSA: Information of pageid and offset copied at the last time on the target node by the replication log copy process. There will be a delay in copying logs as much as difference with the EOF LSA value of "Active Info." and with the Append LSA value of "Copied Active Info."
  • Append LSA: Information of pageid and offset written at the last time on the disk by the replication log copy process. This value can be less than or equal to EOF LSA. There will be a delay in copying logs as much as difference between the EOF LSA value of "Copied Active Info." and this value.
  • HA server state: Status of a database server process which replication log copy process receives logs from. For details on status, see Servers.
• Active Info.
  • DB name: Name of a database whose node was configured in the -r option.
  • DB creation time: Database creation time of a node that is configured in the -r option.
  • EOF LSA: The last information of pageid and offset of a database transaction log of a node that is configured in the -r option. There will be a delay in copying logs as much as difference between the EOF LSA value of "Copied Active Info." and this value.
  • Append LSA: Information of pageid and offset written at the last time on the disk by the database whose node was configured in the -r option.
  • HA server state: The server status of a database server whose node was configured in the -r option.
• Delay in Copying Active Log
  • Delayed log page count: the count of transaction log pages which the copy is delayed.
  • Estimated Delay: the expected time which the logs copying is completed.
• Delay in Applying Copied Log
  • Delayed log page count: the count of transaction log pages which the application is delayed.
  • Estimated Delay: the expected time which the logs applying is completed.

When you run this command in replica node, if "ha_replica_delay=30s" is specified in cubrid.conf, the following information is printed out additionally.

 *** Replica-specific Info. ***
Deliberate lag                 : 30 second(s)
Last applied log record time   : 2013-06-20 11:20:10

Each item of the status information is as below.

• Replica-specific Info.
  • Deliberate lag: delayed time a user defined by ha_replica_delay parameter
  • Last applied log record time: the time where the replication log of being applied in the replica node recently was actually applied in the master node.

When you run this command in replica node, if "ha_replica_delay=30s" and "ha_replica_time_bound=2013-06-20 11:31:00" are specified in cubrid.conf, "ha_replica_delay=30s" is ignored and the following information is printed out additionally.

 *** Replica-specific Info. ***
Last applied log record time   : 2013-06-20 11:25:17
Will apply log records up to   : 2013-06-20 11:31:00

Each item of the status information is as below.

• Replica-specific Info.
  • Last applied log record time: the time where the replication log of being applied in the replica node recently was actually applied in the master node.
  • Will apply log records up to: the replica node will apply the master node's logs replicated up to this time.

When applylogdb stops the replication after the time of being specified by ha_replica_time_bound, the error message which is printed out in the file, $CUBRID/log/db-name@local-node-name_applylogdb_db-name_remote-node-name.err is as below.

Time: 06/20/13 11:51:05.549 - ERROR *** file ../../src/transaction/log_applier.c, line 7913 ERROR CODE = -1040 Tran = 1, EID = 3
HA generic: applylogdb paused since it reached a log record committed on master at 2013-06-20 11:31:00 or later.
Adjust or remove ha_replica_time_bound and restart applylogdb to resume.

cubrid changemode¶

This utility is used to check and change the server status of CUBRID HA.

cubrid changemode [options] <database-name@node-name>

• database-name@node-name: Specifies the name of a server to be checked or changed and separates each node name by using @. If [options] is omitted, server status is displayed.

The following shows [options] used in cubrid changemode.

-m, --mode=MODE¶

Changes the server status. You can enter one of the following:

standby, maintenance or active.

• You can change a mode as maintenance if a server's status is standby.
• You can change a mode as standby if a server's status is maintenance.
• You can change a mode as active if a server's status is to-be-active. However, this should be used together with --force option. See the below descrioption regarding --force option.

-f, --force¶

Configures whether or not to forcibly change the server status. This option must be configured if you want to change the server status from to-be-active to active. |

If it is not configured, the status will not be changed to active. Forcibly change may cause data inconsistency among replication nodes; so it is not recommended. |

-t, --timeout=SECOND¶

The default is 5(seconds). Configures the waiting time for the normal completion of the transaction that is being processed when the node status switches from standby to maintenance.

If the transaction is still in progress beyond the configured time, it will be forced to terminate and switch to maintenance status; if all transactions have completed normally within the configured time, it will switch to maintenance status immediately.

Status Changeable

This table shows changeable modes depending on current status.

* When the server status is to-be-active, forcibly change may cause data inconsistency among replication nodes. It is not recommended if you are not skilled enough.

Example

The following example shows how to switch the testdb server status in the localhost node to maintenance. The waiting time for all transactions in progress to complete normally is 5 seconds, which is the default value for the -t option. If all transactions are complete within this time limit, the status will be switched immediately. However, if there are transactions still being processed after this time limit, they will be rolled back before changing the status.

$ cubrid changemode -m maintenance testdb@localhost
The server 'testdb@localhost''s current HA running mode is maintenance.

The following example shows how to retrieve status of the testdb server in the localhost node.

$ cubrid changemode testdb@localhost
The server 'testdb@localhost''s current HA running mode is active.

Monitoring CUBRID Manager HA¶

CUBRID Manager is a dedicated CUBRID database management tool that provides the CUBRID database management and query features in a GUI environment. CUBRID Manager provides the HA dashboard, which shows the relationship diagram for the CUBRID HA group and server status. For details, see CUBRID Manager manual.

Basic Structure of HA¶

The most basic structure of CUBRID HA consists of one master node and one slave node.

The default configuration is one master node and one slave node. To distribute the write load, a multi-slave node or load-distributed configuration is recommended. In addition, to access a specific node such as a slave node or replica node in read-only mode, configure the Read Only broker or the Preferred Host Read Only broker. For details about broker configuration, see Duplexing Brokers.

An Example of Node Configuration

You can configure each node in the basic structure of HA as shown below:

• node A (master node)

  • Configure the ha_mode of the cubrid.conf file to on.

    ha_mode=on
    

  • The following example shows how to configure cubrid_ha.conf:

    ha_port_id=59901
    ha_node_list=cubrid@nodeA:nodeB
    ha_db_list=testdb
    

• node B (slave node): Configure this node in the same manner as node A.

For the databases.txt file of a broker node, it is necessary to configure the list of hosts configured as HA in db-host according to their priority. The following example shows the databases.txt file.

#db-name    vol-path                  db-host       log-path                   lob-base-path
testdb     /home/cubrid/DB/testdb     nodeA:nodeB   /home/cubrid/DB/testdb/log file:/home/cubrid/DB/testdb/lob

The cubrid_broker.conf file can be set in a variety of ways according to configuration of the broker. It can also be configured as separate equipment with the databases.txt file.

The example below shows that the RW broker is set in each node, and node A and node B have the same value.

[%RW_broker]
...

# Broker mode setting parameter
    ACCESS_MODE             =RW

Connection Configuration of Applications

See JDBC Configuration, CCI Configuration, and PHP Configuration in Environment Configuration.

Remark

The moving path of a transaction log in these configurations is as follows:

Multiple-Slave Node Structure¶

In multiple-slave node structure, there is one master node and several slave nodes to improve the service availability of CUBRID.

Because replication log copy process and replication log reflection process are running at all nodes in the CUBRID HA group, a load of copying replication log occurs. Therefore, all nodes in the CUBRID HA group have high network and disk usage.

Because there are many nodes with HA enabled, read and write services never fail as long as a single node is alive.

In the multiple-slave node structure, the node becoming a master node when failover occurs is determined by the order specified in ha_node_list. If the value of ha_node_list is node1:node2:node3 and the master node is node A, node B will become a new master node when the master node fails.

An Example of Node Configuration

You can configure each node in the basic structure of HA as shown below:

• node A (master node)

  • Configure the ha_mode of the cubrid.conf file to on.

    ha_mode=on
    

  • The following example shows how to configure cubrid_ha.conf:

    ha_port_id=59901
    ha_node_list=cubrid@nodeA:nodeB:nodeC
    ha_db_list=testdb
    ha_copy_sync_mode=sync:sync:sync
    

• node B, node C (slave node): Configure this node in the same manner as node A.

You must enter the list of hosts configured in HA in order of priority in the databases.txt file of a broker node. The following is an example of the databases.txt file.

#db-name   vol-path                   db-host             log-path                   lob-base-path
testdb     /home/cubrid/DB/testdb     nodeA:nodeB:nodeC   /home/cubrid/DB/testdb/log file:/home/cubrid/DB/testdb/lob

The cubrid_broker.conf file can be set in a variety of ways according to configuration of the broker. It can also be configured as separate equipment with the databases.txt file. In this example, the RW broker is configured in node A, node B, and node C.

The following is an example of the databases.txt file in node A, node B, and node C.

[%RW_broker]
...

# Broker mode setting parameter
ACCESS_MODE             =RW

Connection Configuration of Applications

Connect the application to access to the broker of node A, node B, or node C.

Connection connection = DriverManager.getConnection(
    "jdbc:CUBRID:nodeA:33000:testdb:::?charSet=utf-8&altHosts=nodeB:33000,nodeC:33000", "dba", "");

For details, see JDBC Configuration, CCI Configuration, and PHP Configuration in Environment Configuration.

Remark

The moving path of a transaction log in these configurations is as follows:

Load Balancing Structure¶

The load balancing structure increases the availability of the CUBRID service by placing several nodes in the HA configuration (one master node and one slave node) and distributes read-load.

Because the replica nodes receive replication logs from the nodes in the HA configuration and maintain the same data, and because the nodes in the HA configuration do not receive replication logs from the replica nodes, its network and disk usage rate is lower than that of the multiple-slave structure.

Because replica nodes are not included in the HA structure, they provide read service without failover, even when all other nodes in the HA structure fail.

An Example of Node Configuration

You can configure each node in load balancing structure as shown below:

• node A (master node)

  • Configure the ha_mode of the cubrid.conf file to on.

    ha_mode=on
    

  • The following example shows how to configure cubrid_ha.conf:

    ha_port_id=59901
    ha_node_list=cubrid@nodeA:nodeB
    ha_replica_list=cubrid@nodeC:nodeD:nodeE
    ha_db_list=testdb
    ha_copy_sync_mode=sync:sync
    

• node B (slave node): Configure this node in the same manner as node A.

• node C, node D, node E (replica node)

  • Configure the ha_mode of the cubrid.conf file to replica.

    ha_mode=replica
    

  • You can configure the cubrid_ha.conf file in the same manner as node A.

The cubrid_broker.conf can be set in a variety of ways according to configuration of the broker. It can also be configured as separate equipment with the databases.txt file.

In this example, broker and DB server exist on the same machine; the RW broker is configured in node A and node B; the SO broker with "CONNECT_ORDER=RANDOM" and "PREFERRED_HOSTS=localhost" is configured in node C, node D and node E. each of node C, node D or node E tries to connect to local DB server first because they are set as "PREFERRED_HOSTS=localhost". When it is failed to connect to localhost, it tries to connect to one of db-hosts in databases.txt randomly because they are set as "CONNECT_ORDER=RANDOM".

The following is an example of cubrid_broker.conf in node A and node B.

[%RW_broker]
...

# Broker mode setting parameter
ACCESS_MODE             =RW

The following is an example cubrid_broker.conf in node C, node D and node E.

[%PHRO_broker]
...

# Broker mode setting parameter
ACCESS_MODE             =SO
PREFERRED_HOSTS         =localhost

You must enter the list of DB server hosts in the order so that each broker can be connected appropriate HA or load balancing server in the databases.txt file of a broker node.

The following is an example of the databases.txt file in node A and node B.

#db-name    vol-path                  db-host       log-path             lob-base-path
testdb     /home/cubrid/DB/testdb   nodeA:nodeB   /home/cubrid/DB/testdb/log file:/home/cubrid/CUBRID/testdb/lob

The following is an example of the databases.txt file in node C, node D and node E.

#db-name    vol-path                db-host             log-path                    lob-base-path
testdb     /home/cubrid/DB/testdb   nodeC:nodeD:nodeE   /home/cubrid/DB/testdb/log  file:/home/cubrid/CUBRID/testdb/lob

Connection Configuration of Applications

Connect the application to access in read/write mode to the broker of node A or node B. The following is an example of a JDBC application.

Connection connection = DriverManager.getConnection(
    "jdbc:CUBRID:nodeA:33000:testdb:::?charSet=utf-8&altHosts=nodeB:33000", "dba", "");

Connect the application to access in read-only mode to the broker of node C, node D or node E. The following is an example of a JDBC application. Configure "loadBalance=true" on the URL to connect randomly to the main host and hosts which are specified by altHosts.

Connection connection = DriverManager.getConnection(
    "jdbc:CUBRID:nodeC:33000:testdb:::?charSet=utf-8&loadBalance=true&altHosts=nodeD:33000,nodeE:33000", "dba", "");

For details, see JDBC Configuration, CCI Configuration, and PHP Configuration in Environment Configuration.

Remark

The path of a transaction log in these configurations is as follows:

Multiple-Standby Server Structure¶

Although its node structure has a single master node and a single slave node, many slave nodes from different services are physically configured in a single server.

This structure is for very small services in which the reading load of slave nodes is light. It is strictly for the availability of the CUBRID service. For this reason, when a master node that failed after a failover has been restored, the load must be moved back to the original master node to minimize the load of the server with multiple-slave nodes.

An Example of Node Configuration

You can configure each node in the basic structure of HA as shown below:

• node AM, node AS : Configure them in the same manner.

  • Configure the ha_mode of the cubrid.conf file to on.

    ha_mode=on
    

  • The following example shows how to configure cubrid_ha.conf.

    ha_port_id=10000
    ha_node_list=cubridA@Host1:Host5
    ha_db_list=testdbA1,testdbA2
    ha_copy_sync_mode=sync:sync
    

• node BM, node BS : Configure them in the same manner.

  • Configure the ha_mode of the cubrid.conf file to on.

    ha_mode=on
    

  • The following example shows how to configure cubrid_ha.conf.

    ha_port_id=10001
    ha_node_list=cubridB@Host2:Host5
    ha_db_list=testdbB1,testdbB2
    ha_copy_sync_mode=sync:sync
    

• node CM, node CS : Configure them in the same manner.

  • Configure the ha_mode of the cubrid.conf file to on.

    ha_mode=on
    

  • The following example shows how to configure cubrid_ha.conf.

    ha_port_id=10002
    ha_node_list=cubridC@Host3:Host5
    ha_db_list=testdbC1,testdbC2
    ha_copy_sync_mode=sync:sync
    

• node DM, node DS : Configure them in the same manner.

  • Configure the ha_mode of the cubrid.conf file to on.

    ha_mode=on
    

  • The following is an example of the cubrid_ha.conf configuration.

    ha_port_id=10003
    ha_node_list=cubridD@Host4:Host5
    ha_db_list=testdbD1,testdbD2
    ha_copy_sync_mode=sync:sync
    

Operation Scenario during Read/Write Service¶

The operation scenario written in this page is not affected by read/write services. Therefore, its impact on the services caused by CUBRID operation is very limited. There can be two types of operation scenarios in which failover occurs or it does not occur.

When Failover Does Not Occur

You can perform the following operations without stopping and restarting nodes in CUBRID HA groups.

When Failover Occurs

You must stop nodes in CUBRID HA group and complete operation before performing the following operations.

Operation Scenario during Read Service¶

The operation scenario written in this page is only applied to read service. It is required to allow read service only or dynamically change mode configuration of broker to Read Only. There can be two types of operation scenarios in which failover occurs or it does not occur.

When Failover Does Not Occur

You can perform the following operations without stopping and restarting nodes in CUBRID HA groups.

When Failover Occurs

You must stop nodes in CUBRID HA group and complete operation before performing the following operations.

Operation Scenario after Stopping a Service¶

You must stop all nodes in CUBRID HA group before performing the following operation.

Setting Replica Replication Delay¶

This scenario delays the replication of the master node data in replica node, and stops the replication of the master node data at the specific time, to detect the case which someone deleted the data by mistake and stop the replication at the specified time.

restoreslave¶

cubrid restoreslave is the same as cubrid restoredb which restores a database from a backup but it includes several convenient features when rebuilding a slave or a replica. With cubrid restoreslave, user does not need to manually collect replication-related information from a backup output to create a replication catalog which is stored in db_ha_apply_info. It automatically reads any necessary information from a backup image and an active log and then it adds the relevant replication catalog into db_ha_apply_info. All you need to do is to provide two mandatory options: the state of the node where the backup image was created, and the hostname of the current master node. Please also refer restoredb for more details.

cubrid restoreslave [OPTION] database-name

-s, --source-state=STATE¶

You need to specify the state of the node where the backup image was created. STATE may be 'master', 'slave', or 'replica'

-m, --master-host-name=NAME¶

You need to specify the hostname of the current master node.

--list¶

This option displays information on backup files of a database; restoration procedure is not performed. For further information, see --list of restoredb

-B, --backup-file-path=PATH¶

You can specify the directory where backup files are to be located by using the -B option. For further information, see -B of restoredb

-o, --output-file=FILE¶

You can specify a filename to store output messages. For further information, see -o of restoredb

-u, --use-database-location-path¶

This option restores a database to the path specified in the database location file(databases.txt). For further information, see -u of restoredb

Example Scenarios of Building Replication¶

In this section, we will see various scenarios regarding adding a new node or removing a node during HA service.

• Add a Slave after Stopping a Service

  Add a new slave node after the service stops when operating a database with only a machine.

• Build Another Slave during HA Service

  Add a new slave using the existing slave node in the HA environment which one master and one slave are built.

• Remove a Slave during Service

  Remove a slave when one master and two slaves are built.

• Add a Replica during Service

  Add a new replica node using the existing slave in the environment which a master and a slave are built.

• Rebuild a Slave during Service

  Rebuild the slave using the existing master in the environment which a master, a slave and a replica are built. A new information should be updated by a user in the db_ha_apply_info catalog table.

Now let's see in detail about the above scenarios.

When you want to rebuild only the abnormal node in the HA environment, see Rebuilding Replication Script.

Add a Slave after Stopping a Service¶

Now let's see the case of building a new slave node with a single master node - making a 1:1 master-slave composition.

• nodeA: a master node's host name
• nodeB: a slave node's host name to be added newly

This scenario assumes that the database has been created by below commands. When you run createdb, a locale name and a charset should be the same between a master node and a slave node.

export CUBRID_DATABASES=/home/cubrid/DB
mkdir $CUBRID_DATABASES/testdb
mkdir $CUBRID_DATABASES/testdb/log
cd $CUBRID_DATABASES/testdb
cubrid createdb testdb -L $CUBRID_DATABASES/testdb/log en_US.utf8

The backup file is saved in the $CUBRID_DATABASES/testdb directory by default if the location is not specified.

URL's change in an application is not required if a broker node is composed of a separated machine and if adding a new broker is not considered.

In this scenario, we assume that broker's node is not installed on a separated machine.

With the above in mind, let's work as follows.

0. Configure HA of master node and slave node.

   • Set the $CUBRID/conf/cubrid.conf identically on master and slave nodes.

     [service]
     service=server,broker,manager
     
     # Add the database name to run when starting the service
     server=testdb
     
     [common]
     ...
     
     # Add when configuring the HA (Logging parameters)
     force_remove_log_archives=no
     
     # Add when configuring the HA (HA mode)
     ha_mode=on
     

   • Set the $CUBRID/conf/cubrid_ha.conf identically on master and slave nodes.

     [common]
     ha_port_id=59901
     
     # cubrid is a group name of HA system, nodeA and nodeB are host names.
     ha_node_list=cubrid@nodeA:nodeB
     
     ha_db_list=testdb
     ha_copy_sync_mode=sync:sync
     ha_apply_max_mem_size=500
     

   • Set the $CUBRID_DATABASES/databases.txt identically on master and slave nodes.

     #db-name    vol-path                db-host         log-path                    lob-base-path
     testdb      /home/cubrid/DB/testdb  nodeA:nodeB     /home/cubrid/DB/testdb/log  file:/home/cubrid/DB/testdb/lob
     

   • Set brokers and applications in HA environment.

     For broker setting and adding altHosts into URL, see cubrid_broker.conf.

   • Set locale library identically on master and slave nodes.

     [nodeA]$ scp $CUBRID/conf/cubrid_locales.txt cubrid_usr@nodeB:$CUBRID/conf/.
     [nodeB]$ make_locale.sh -t64
     

   • Create a directory to a slave node.

     [nodeB]$ cd $CUBRID_DATABASES
     [nodeB]$ mkdir testdb
     

   • Create a log directory to a slave node. (create this on the same path with a master node.)

     [nodeB]$ cd $CUBRID_DATABASES/testdb
     [nodeB]$ mkdir log
     

1. Stop a master node, backup a master database.

   [nodeA]$ cubrid service stop
   

   Back up the database of the master node and copy the backup file to the slave node.

   If the location where the backup file will be saved in the master node is not specified, the location is set as the log directory of testdb by default. Copy the backup file to the same location in the slave node. testdb_bk0v000 is the backup volume file and testdb_bkvinf is the backup volume information file.

   [nodeA]$ cubrid backupdb -z -S testdb
   Backup Volume Label: Level: 0, Unit: 0, Database testdb, Backup Time: Thu Apr 19 16:05:18 2012
   [nodeA]$ cd $CUBRID_DATABASES/testdb/log
   [nodeA]$ scp testdb_bk* cubrid_usr@nodeB:$CUBRID_DATABASES/testdb/log
   cubrid_usr@nodeB's password:
   testdb_bk0v000                            100% 6157KB   6.0MB/s   00:00
   testdb_bkvinf                             100%   66     0.1KB/s   00:00
   

2. Restore a database from a slave node.

   Restore the database in the slave node. At this time, the volume path of the master node must be identical to that of the slave node.

   [nodeB]$ cubrid restoredb -B $CUBRID_DATABASES/testdb/log demodb
   

3. Start a master node, start a slave node.

   [nodeA]$ cubrid heartbeat start
   

   After confirming that the master node has been started, start the slave node.

   If nodeA's state is changed from registered_and_to_be_active to registered_and_active, it means that the master node has been successfully started.

   [nodeA]$ cubrid heartbeat status
   @ cubrid heartbeat status
   
    HA-Node Info (current nodeA, state master)
      Node nodeB (priority 2, state unknown)
      Node nodeA (priority 1, state master)
   
    HA-Process Info (master 123, state master)
   
      Applylogdb testdb@localhost:/home1/cubrid/DB/tdb01_nodeB (pid 234, state registered)
      Copylogdb testdb@nodeB:/home1/cubrid/DB/tdb01_nodeB (pid 345, state registered)
      Server tdb01 (pid 456, state registered_and_to_be_active)
   
   [nodeB]$ cubrid heartbeat start
   

   Confirm that the HA configurations of the master node and the slave node are successfully running

   [nodeA]$ csql -u dba testdb@localhost -c"create table tbl(i int primary key);insert into tbl values (1),(2),(3)"
   
   [nodeB]$ csql -u dba testdb@localhost -c"select * from tbl"
   
               i
   =============
               1
               2
               3
   

Build Another Slave during HA Service¶

The following is a scenario to add a new slave from an existing slave during a HA service. "Master:Slave" will be from 1:1 to 1:2.

• nodeA: a master node's host name
• nodeB: a slave node's host name
• nodeC: a slave node's host name to be added

You can use an existing master or slave if you want to add a new slave during HA service. In this scenario, let's add a new slave by using an existing slave because we assume that slave's disk I/O is less than master's.

export CUBRID_DATABASES=/home/cubrid/DB
mkdir $CUBRID_DATABASES/testdb
mkdir $CUBRID_DATABASES/testdb/log
cd $CUBRID_DATABASES/testdb
cubrid createdb testdb -L $CUBRID_DATABASES/testdb/log en_US.utf8

0. HA configuration of nodeC

   • Set $CUBRID/conf/cubrid.conf of nodeC identically with nodeB's.

     "force_remove_log_archives=no" should be set not to let no-replicated logs be deleted during HA service.

     [service]
     service=server,broker,manager
     
     [common]
     ...
     
     # Add when configuring the HA (Logging parameters)
     force_remove_log_archives=no
     
     # Add when configuring the HA (HA mode)
     ha_mode=on
     

   • Set $CUBRID/conf/cubrid_ha.conf of nodeC identically with nodeB's.

     However, add nodeC into ha_node_list, and add one more "sync" into ha_copy_sync_mode.

     [common]
     ha_port_id=59901
     
     # cubrid is a group name of HA system, nodeA, nodeB, and nodeC are host names.
     ha_node_list=cubrid@nodeA:nodeB:nodeC
     
     ha_db_list=testdb
     ha_copy_sync_mode=sync:sync:sync
     ha_apply_max_mem_size=500
     

   • Set a locale library of nodeC identically with that of nodeA or nodeB.

     [nodeB]$ scp $CUBRID/conf/cubrid_locales.txt cubrid_usr@nodeC:$CUBRID/conf/.
     [nodeC]$ make_locale.sh -t64
     

   • Add nodeA, nodeB, and nodeC into db-host of $CUBRID_DATABASES/databases.txt in nodeC.

     #db-name     vol-path                   db-host             log-path                        lob-base-path
     testdb       /home/cubrid/DB/testdb     nodeA:nodeB:nodeC   /home/cubrid/DB/testdb/log      file:/home/cubrid/DB/testdb/lob
     

     db-host of databases.txt is related to the access order to the database server by a broker; therefore, you can change the value as you want. For example, you can write only nodeA:nodeB or localhost into db-host. However, by considering the local machine access, (e.g. csql -u dba testdb@localhost) you should add at least localhost or the name of localhost into db-host.

1. Apply configuration for preventing from deleting replication logs of nodeB during restoring after backup

   Replication logs of nodeB can be added if service continues after restoring a database from the nodeB backup file to nodeC. By the way, slave's replication logs can be deleted before being copied to nodeC by configuration. To prevent this case, you can configure system parameters as follows.

   • Change the value of ha_copy_log_max_archives bigger without restarting nodeB's database.

     From backup of nodeB, transactions which have been executed after backup of nodeB should be kept; then transactions after backup can be applied to nodeC.

     Therefore, added replication logs of nodeB after backup should be kept.

     Here, let's assume the number of replication logs as 5, which can sufficiently save executed transactions during building nodeC.

     Edit $CUBRID/conf/cubrid_ha.conf

     ha_copy_log_max_archives=5
     

     Apply the change of the parameter ha_copy_log_max_archives to the applylogdb process of nodeB.

     [nodeB]$ cubrid heartbeat applylogdb stop testdb nodeA
     [nodeB]$ cubrid heartbeat applylogdb start testdb nodeA
     

     Note

     If you build nodeC with only nodeA, a master node, to change the value of ha_copy_log_max_archives from nodeB is needless because copying replication logs from nodeB is needless.

     Instead, you should apply the changed log_max_archives to nodeA.

     The value of log_max_archives parameter of cubrid.conf can be changed during service by using "SET SYSTEM PARAMETERS" syntax.

     [nodeA]$ csql -u dba -c "SET SYSTEM PARAMETERS 'log_max_archives=5'" testdb@localhost
     

2. Backup from nodeB, restore into nodeC

   • Backup a database from nodeB.

     [nodeB]$ cd $CUBRID_DATABASES/testdb/log
     [nodeB]$ cubrid backupdb --sleep-msecs=10 -C -o output.txt testdb@localhost
     

     Note

     --sleep-msecs is an option to set the rest time in each cycle when 1MB is written in a backup file.The unit is millisecond. Consider setting this option if the disk I/O overhead of a machine to back up is severe; however, it is recommended to set this value as small as possible and do backup when backup overhead is small enough because if this value is bigger, backup time takes longer.

     If there is no I/O overhead on nodeB, you can omit this option.

     On the file specified in -o option, the backup result information is written.

   • Copy backup files to nodeC.

     [nodeB]$
     [nodeB]$ scp -l 131072 testdb_bk* cubrid_usr@nodeC:$CUBRID_DATABASES/testdb/log
     

     Note

     scp command's -l option is an option to control the amount of copying file; when you copy files between nodes, use this value properly as considering I/O overhead. The unit is Kbits. 131072 is 16MB.

   • Restore the database into nodeC.

     At this time, the volume path of the slave must be the same with the master.

     [nodeC]$ cubrid restoredb -B $CUBRID_DATABASES/testdb/log testdb
     

3. Copy the active log of nodeA into nodeC

   In nodeA's active log, there exists a number information of the archive log which is recently created. This information is needed for archive logs, which has been created after the time in which the active log had been copied, to be automatically copied by copylogdb process after the start of HA service of nodeC; therefore, the user just needs to manually copy archive logs which has been created before copying an active log.

   • Start cub_master which manages HA connection in nodeC.

     [nodeC]$ cub_master
     

   • Start copylogdb process for nodeA.

     [nodeC]$ cubrid heartbeat copylogdb start testdb nodeA
     

   • Check if nodeA's active log was copied.

     To copy the current active log means to obtain the number information of the last archive log. Archive logs after this time are automatically copied after the process [5. Start HA server in nodeC].

     [nodeC]$ cd $CUBRID_DATABASES/testdb_nodeA
     [nodeC]$ ls
     testdb_lgar_t  testdb_lgat  testdb_lgat__lock
     

   • Stop copylogdb process for nodeA on nodeC.

     [nodeC]$ cubrid heartbeat copylogdb stop testdb nodeA
     

   • Stop cub_master of nodeC.

     [nodeC]$ cubrid service stop
     

4. Copy required log files after restoring a database into nodeC

   • Copy all replicated archive logs of nodeB into nodeC.

     [nodeC]$ cd $CUBRID_DATABASES/testdb_nodeA
     [nodeC]$ scp -l 131072 cubrid@nodeB:$CUBRID_DATABASES/testdb_nodeA/testdb_lgar0* .
     

     In this process, please note that all replicated archive logs which are required for replication should exist.

     Note

     scp command's -l option is an option to control the amount of copying file; when you copy files between nodes, use this value properly as considering I/O overhead. The unit is Kbits. 131072 is 16MB.

     Note

     Copy files of which the names end with a number. Here, we copied with "testdb_lgar0*" because numbers of names of all replication log files are started with 0.

     Note

     Preparation plan when the required replication logs are already deleted

     Replication logs which are created between backup time and active log creation time can be deleted if the value of ha_copy_log_max_archives is not large enough; in this case, an error can occur when running the below process, [5. Start HA service on nodeC].

     When an error occurs, start HA service of nodeC by adding archive logs in master if the required logs exist in master node. For example, you can copy an archive log #1 from master if there are archive logs #1, #2, and #3 in master; and there are only replicated archive logs #2, and #3 in slave and #1 is required additionally. However, the case in which master's archive log files are left more can happen only when the value of log_max_archives of master is big enough.

     If there is no wanted number's archive log in master's archive logs, change the value of ha_copy_log_max_archives big enough and do backup-and-restore procedure from the beginning.

     Note

     Check if required replication logs are deleted

     You can check if requried replication logs are deleted or not by watching the testdb_lginf file.

     1. If this is the case of restoring a database from a slave backup file, you can check what pages are required, from the page of master by watching the value of required_lsa_pageid in the db_ha_apply_info catalog table.
     2. Watch the contents of the testdb_lginf file located in $CUBRID_DATABASES/testdb/log directory from bottom to top.

     Time: 03/16/15 17:44:23.767 - COMMENT: CUBRID/LogInfo for database /home/cubrid/DB/databases/testdb/testdb
     Time: 03/16/15 17:44:23.767 - ACTIVE: /home/cubrid/DB/databases/testdb/log/testdb_lgat 1280 pages
     Time: 03/16/15 17:54:40.892 - ARCHIVE: 0 /home/cubrid/DB/databases/testdb/log/testdb_lgar000 0 1277
     Time: 03/16/15 17:57:29.451 - COMMENT: Log archive /home/cubrid/DB/databases/testdb/log/testdb_lgar000 is not needed any longer unless a database media crash occurs.
     Time: 03/16/15 18:03:08.166 - ARCHIVE: 1 /home/cubrid/DB/databases/testdb/log/testdb_lgar001 1278 2555
     Time: 03/16/15 18:03:08.167 - COMMENT: Log archive /home/cubrid/DB/databases/testdb/log/testdb_lgar000, which contains log pages before 2556, is not needed any longer by any HA utilities.
     Time: 03/16/15 18:03:29.378 - COMMENT: Log archive /home/cubrid/DB/databases/testdb/log/testdb_lgar001 is not needed any longer unless a database media crash occurs.
     

     When the contents of testdb_lginf is written as the above, the last two numbers in "Time: 03/16/15 17:54:40.892 - ARCHIVE: 0 /home/cubrid/DB/databases/testdb/log/testdb_lgar000 0 1277" are the start and end IDs of pages which this file store.

     For example, if the page ID at backup is 2300, we can see this ID is between no. 1278 and no. 2555 by watching "testdb_lgar001 1278 2555"; therefore, we can know that the archive logs of master (or replicated archive logs of slave) are required from no. 1 when restoring a database.

5. Start HA service on nodeC

   [nodeC]$ cubrid heartbeat start
   

6. Change HA configuration of nodeA and nodeB

   • Turn back the value of ha_copy_log_max_archive of nodeB.

     [nodeB]$ vi cubrid_ha.conf
     
     ha_copy_log_max_archives=1
     

   • Apply the changed value of ha_copy_log_max_archives to the applylogdb process of nodeB.

     [nodeB]$ cubrid heartbeat applylogdb stop testdb nodeA
     [nodeB]$ cubrid heartbeat applylogdb start testdb nodeA
     

   • In cubrid_ha.conf, add nodeC into ha_node_list, and add one "sync" more to nodeA and nodeB.

     $ cd $CUBRID/conf
     $ vi cubrid_ha.conf
     
     ha_node_list=cubrid@nodeA:nodeB:nodeC
     ha_copy_sync_mode=sync:sync:sync
     

   • Apply changed ha_node_list.

     [nodeA]$ cubrid heartbeat reload
     [nodeB]$ cubrid heartbeat reload
     

   • Start coplylogdb and applylogdb for nodeC.

     [nodeA]$ cubrid heartbeat repl start nodeC
     [nodeB]$ cubrid heartbeat repl start nodeC
     

7. Add a broker and add this broker's host name into altHosts of application URL

   Add a broker if needed, and add this broker's host name into altHosts, the property of application URL to access this broker.

   If you are not considering to add a broker, you do not need to do this job.

   • From all of nodeA and nodeB, add nodeC into db-host of $CUBRID_DATABASES/databases.txt.

   #db-name     vol-path                   db-host             log-path                        lob-base-path
   testdb       /home/cubrid/DB/testdb     nodeA:nodeB:nodeC   /home/cubrid/DB/testdb/log      file:/home/cubrid/DB/testdb/lob
   

   db-host of databases.txt is a configuration parameter related to the order of access from a broker to database servers; therefore, you can change as you want.For example, you can write only "nodeA:nodeB" or "localhost" into db-host. However, if you access database from a local machine (e.g. csql -u dba testdb@localhost), you should include "localhost" or the localhost name into db-host.

Remove a Slave during Service¶

Let's remove a slave when the HA environment is composed of "master:slave = 1:2".

• nodeA: a master node's host name
• nodeB: a slave node's host name
• nodeC: a slave node's host name to be removed

1. Stop the service of nodeC

   $ cubrid heartbeat stop     # Processes related to heartbeat (applylogdb, copylogdb, cub_server) are stopped.
   $ cubrid service stop       # Processes such as cub_master, cub_broker, and cub_manager are stopped.
   

2. Remove nodeC from the configuration of nodeA, and nodeB

• Remove nodeC in ha_node_list from cubrid_ha.conf of nodeA and nodeB; remove one "sync" in ha_copy_sync_mode.

  $ vi cubrid_ha.conf
  
  ha_node_list=cubrid@nodeA:nodeB
  ha_copy_sync_mode=sync:sync
  

• Apply the changed ha_node_list.

  Run "reload" command at the master, nodeA.

  [nodeA]$ cubrid heartbeat reload
  

  Run "stop" and "start" commands at the slave, nodeB.

  [nodeB]$ cubrid heartbeat stop
  [nodeB]$ cubrid heartbeat start
  

• Stop coplylogdb and applylogdb for nodeC.

  [nodeA]$ cubrid heartbeat repl stop nodeC
  [nodeB]$ cubrid heartbeat repl stop nodeC
  

• Remove replication logs about nodeC from nodeA and nodeB.

  $ cd $CUBRID_DATABASES
  $ rm -rf testdb_nodeC
  

• Remove replication information about nodeC from nodeA and nodeB.

  You can run DELELTE operation from a slave node only if you add --sysadm and --write-on-standby options on running csql.

  $ csql -u dba --sysadm --write-on-standby testdb@localhost
  
  sysadm> DELETE FROM db_ha_apply_info WHERE copied_log_path='/home/cubrid/DB/databases/testdb_nodeC';
  

  copied_log_path is the path in which replication log files are stored.

3. Remove nodeC from broker's configuration

   • Remove nodeC if this exists in db-host of databases.txt from nodeA and nodeB.

     $ cd $CUBRID_DATABASES
     $ vi databases.txt
     
     #db-name     vol-path                   db-host             log-path                        lob-base-path
     testdb       /home/cubrid/DB/testdb     nodeA:nodeB         /home/cubrid/DB/testdb/log      file:/home/cubrid/DB/testdb/lob
     

   • Restart brokers of nodeA and nodeB.

     $ cubrid broker restart
     

     Note

     You don't need to hurry to restart these brokers because nodeC is already stopped and an application will not access the nodeC database. If you restart these brokers, you should consider that the applications connected to these brokers can be disconnected. Moreover, if multiple brokers with the same configuration exist, one broker can substitute others; therefore, restart a broker one by one.

     For example, on the above, nodeA and nodeB are the same configuration; therefore, you can restart nodeB first then restart nodeA.

Add a Replica during Service¶

Now let's add a replica when HA environment is set as "master:slave=1:1". The beginning database setting is the same as the setting in Build an initial database of Build Another Slave during HA Service.

• nodeA: a master node's host name
• nodeB: a slave node's host name
• nodeC: a replica node's host name to be added newly

0. Configure HA parameter of nodeC

   • Set nodeC's $CUBRID/conf/cubrid.conf as the same with nodeB except for ha_mode.

     Set "ha_mode=replica".

     [service]
     service=server,broker,manager
     
     [common]
     
     # Add this when doing HA configuration (HA mode)
     ha_mode=replica
     

     Note

     Replica node's transaction logs are not used to replication; therefore, whatever the value of force_remove_log_archives is set to, it works as yes.

   • Configure nodeC's $CUBRID/conf/cubrid_ha.conf

     Add nodeC into ha_replica_list.

     [common]
     ha_port_id=59901
     
     # cubrid is a group name of HA system, nodeA, nodeB, and nodeC are host names.
     ha_node_list=cubrid@nodeA:nodeB
     
     ha_replica_list=cubrid@nodeC
     
     ha_db_list=testdb
     ha_copy_sync_mode=sync:sync
     ha_apply_max_mem_size=500
     

   • Set a locale library of nodeC identically with that of nodeA or nodeB.

     [nodeB]$ scp $CUBRID/conf/cubrid_locales.txt cubrid_usr@nodeC:$CUBRID/conf/.
     [nodeC]$ make_locale.sh -t64
     

   • Add nodeC into db-host of $CUBRID_DATABASES/databases.txt in nodeC.

     #db-name     vol-path                   db-host             log-path                        lob-base-path
     testdb       /home/cubrid/DB/testdb     nodeC   /home/cubrid/DB/testdb/log      file:/home/cubrid/DB/testdb/lob
     

     db-host of databases.txt is related to the access order to the database server by a broker; therefore, you can change the value as you want. For example, you can write only nodeA:nodeB or localhost into db-host. However, by considering the local machine access, (e.g. csql -u dba testdb@localhost) you should add at least localhost or the name of localhost into db-host.

1. Apply configuration for preventing from deleting replication logs of nodeB during restoring after backup

   nodeB's replication logs can be added after restoring if a service keeps going during restoring on nodeC after backup of nodeB; at this time, some replication logs which are needed for nodeC can be deleted by the configuration. To protect this, configure nodes as following.

   • Change the value of ha_copy_log_max_archives larger without restarting a database on nodeB.

     To apply transactions after backup time to nodeC, transactions should be kept until the nodeC's data which restore nodeB's backup data become the same with master(nodeA)'s.

     Therefore, nodeB's replication logs which are added after backup should be kept.

     Here, let's assume the number of replication log files which can sufficiently store transactions executed during nodeC's building as 5.

     Edit $CUBRID/conf/cubrid_ha.conf

     ha_copy_log_max_archives=5
     

     Apply the change of ha_copy_log_max_archives to nodeB's applylogdb process.

     [nodeB]$ cubrid heartbeat applylogdb stop testdb nodeA
     [nodeB]$ cubrid heartbeat applylogdb start testdb nodeA
     

     Note

     If you build nodeC with nodeA, a master node only, copying replication logs from nodeB is not needed; therefore, changing the value of ha_copy_log_max_archives from nodeB is also needless.

     Instead, it requires to apply a changed value of log_max_archives to nodeA.

     The value of log_max_archives parameter in cubrid.conf can be changed during service with the "SET SYSTEM PARAMETERS" syntax.

     [nodeA]$ csql -u dba -c "SET SYSTEM PARAMETERS 'log_max_archives=5'" testdb@localhost
     

2. Backup nocdB and restore on nodeC

   • Backup a database from nodeB.

     [nodeB]$ cd $CUBRID_DATABASES/testdb/log
     [nodeB]$ cubrid backupdb --sleep-msecs=10 -C -o output.txt testdb@localhost
     

     Note

     Please refer to the explanation of --sleep-msecs within Build Another Slave during HA Service.

   • Copy backup files to nodeC.

     [nodeB]$
     [nodeB]$ scp -l 131072 testdb_bk* cubrid_usr@nodeC:$CUBRID_DATABASES/testdb/log
     

     Note

     scp command's -l option is an option to control the amount of copying file; when you copy files between nodes, use this value properly as considering I/O overhead. The unit is Kbits. 131072 is 16MB.

   • Restore the database into nodeC.

     At this time, volume paths between a master and a slave should be the same.

     [nodeC]$ cubrid restoredb -B $CUBRID_DATABASES/testdb/log testdb
     

3. Copy nodeA's active log to nodeC

   In nodeA's active log, there exists a number information of the archive log which is recently created. This information is needed for archive logs, which has been created after the time in which the active log had been copied, to be automatically copied by copylogdb process after the start of HA service of nodeC; therefore, the user just needs to manually copy archive logs which has been created before copying an active log.

   • Start cub_master, which manages HA connections, on nodeC.

     [nodeC]$ cub_master
     

   • Start copylogdb process for nodeA.

     [nodeC]$ cubrid heartbeat copylogdb start testdb nodeA
     

   • Check if nodeA's active log was copied.

     To copy the current active log means to obtain the number information of the last archive log. Archive logs after this time are automatically copied after the process [5. Start HA server in nodeC].

     [nodeC]$ cd $CUBRID_DATABASES/testdb_nodeA
     [nodeC]$ ls
     testdb_lgar_t  testdb_lgat  testdb_lgat__lock
     

   • Stop copylogdb process for nodeA on nodeC.

     [nodeC]$ cubrid heartbeat copylogdb stop testdb nodeA
     

   • Stop cub_master of nodeC.

     [nodeC]$ cubrid service stop
     

4. Copy required log files after restoring a database into nodeC

   • Copy all replicated archive logs of nodeB into nodeC.

     [nodeC]$ cd $CUBRID_DATABASES/testdb_nodeA
     [nodeC]$ scp -l 131072 cubrid@nodeB:$CUBRID_DATABASES/testdb_nodeA/testdb_lgar0* .
     

     In this process, please note that all replicated archive logs which are required for replication should exist.

     Note

     scp command's -l option is an option to control the amount of copying file; when you copy files between nodes, use this properly as considering I/O overhead. The unit is Kbits. 131072 is 16MB.

     Note

     Copy files of which the names end with a number. Here, we copied with "testdb_lgar0*" because numbers of the names of all replication log files are started with 0.

     Note

     Replication logs which are created between backup time and active log creation time can be deleted if the value of ha_copy_log_max_archives is not large enough; in this case, an error can occur when running the below process, [5. Start HA service on nodeC].

     When an error occurs, take actions by referring to Preparation plan when the required replication logs are already deleted described on the above.

5. Start HA service on nodeC

   [nodeC]$ cubrid heartbeat start
   

6. Change HA configuration of nodeA and nodeB

   • Turn back ha_copy_log_max_archive of nodeB.

     [nodeB]$ vi cubrid_ha.conf
     
     ha_copy_log_max_archives=1
     

   • Apply the changed value of ha_copy_log_max_archives to the applylogdb process of nodeB.

     [nodeB]$ cubrid heartbeat applylogdb stop testdb nodeA
     [nodeB]$ cubrid heartbeat applylogdb start testdb nodeA
     

   • In cubrid_ha.conf, add nodeC into ha_node_list, and add one "sync" more to nodeA and nodeB.

     $ cd $CUBRID/conf
     $ vi cubrid_ha.conf
     
     ha_replica_list=cubrid@nodeC
     

   • Apply changed ha_node_list.

     [nodeA]$ cubrid heartbeat reload
     [nodeB]$ cubrid heartbeat reload
     

     At this time, coplylogdb and applylogdb processes for nodeC on nodeA and nodeB are needless to start because nodeC is a replica node.

7. Add a broker and add this broker's host name into altHosts of application URL

   Add a broker if needed, and add this broker's host name into altHosts, the property of application URL to access this broker.

   If you are not considering to add a broker, you do not need to do this job.

   • From all of nodeA and nodeB, add nodeC into db-host of $CUBRID_DATABASES/databases.txt.

   #db-name     vol-path                   db-host             log-path                        lob-base-path
   testdb       /home/cubrid/DB/testdb     nodeA:nodeB:nodeC   /home/cubrid/DB/testdb/log      file:/home/cubrid/DB/testdb/lob
   

   db-host of databases.txt is a configuration parameter related to the order of access from a broker to database servers; therefore, you can change as you want. For example, you can write only "nodeA:nodeB" or "localhost" into db-host. However, if you access database from a local machine (e.g. csql -u dba testdb@localhost), you should include "localhost" or the localhost name into db-host.

Rebuild a Slave during Service¶

Now let's see the case of rebuilding a existing slave node during a service in a composition of "master:slave:relica = 1:1:1". The beginning database setting is the same as the setting in Build an initial database of Build Another Slave during HA Service.

• nodeA: a host name of a master node
• nodeB: a host name of a slave node to be rebuilt
• nodeC: a host name of a replica node

1. Stop nodeB's service and remove this database

   After stopping nodeB's service, remove nodeB's database volumes and nodeB's replication logs located in nodeA and nodeC.

   • Stop nodeB's service.

     [nodeB]$ cubrid heartbeat stop
     [nodeB]$ cubrid service stop
     

   • Remove nodeB's database volumes and replication logs.

     [nodeB]$ cd $CUBRID_DATABASES
     [nodeB]$ rm testdb/*
     [nodeB]$ rm testdb/log/*
     
     [nodeB]$ rm -rf testdb_nodeA
     

   • Stop log replication processes of nodeB on nodeA and nodeC.

     [nodeA]$ cubrid heartbeat repl stop testdb nodeB
     [nodeC]$ cubrid heartbeat repl stop testdb nodeB
     

   • Remove replication logs for nodeB from nodeA and nodeC.

     [nodeA]$ rm -rf $CUBRID_DATABASES/testdb_nodeB
     [nodeC]$ rm -rf $CUBRID_DATABASES/testdb_nodeB
     

2. Remove HA catalog table's data, restore nodeB's database from nodeA's backup, and add data to HA catalog table.

   • Delete the HA catalog table, db_ha_apply_info's records.

     Delete all records of db_ha_apply_info of nodeB to initialize.

     [nodeB]$ csql --sysadm --write-on-standby -u dba -S testdb
     csql> DELETE FROM db_ha_apply_info;
     

     Delete db_ha_apply_info data for nodeB from nodeA and nodeC.

     [nodeA]$ csql --sysadm -u dba testdb@localhost
     csql> DELETE FROM db_ha_apply_info WHERE copied_log_path-='/home/cubrid/DB/databases/testdb_nodeB'
     
     [nodeC]$ csql --sysadm --write-on-standby -u dba testdb@localhost
     csql> DELETE FROM db_ha_apply_info WHERE copied_log_path-='/home/cubrid/DB/databases/testdb_nodeB'
     

   • Backup a database from nodeA.

     [nodeA]$ cd $CUBRID_DATABASES/testdb/log
     [nodeA]$ cubrid backupdb --sleep-msecs=10 -C -o output.txt testdb@localhost
     

     Note

     Please refer to --sleep-msecs of Build Another Slave during HA Service.

   • Copy backup files to nodeB.

     [nodeA]$
     [nodeA]$ scp testdb_bk* cubrid_usr@nodeB:$CUBRID_DATABASES/testdb/log
     

     Note

     scp command's -l option is an option to control the amount of copying file; when you copy files between nodes, use this properly as considering I/O overhead. The unit is Kbits. 131072 is 16MB.

   • Restore the database into nodeB.

     At this time, the volume path of the master node must be identical to that of the slave node.

     [nodeB]$ cubrid restoredb -B $CUBRID_DATABASES/testdb/log testdb
     

   • Add a replication information record for nodeA to nodeB's db_ha_apply_info.

     Get an update information of db_ha_apply_info from output.txt file, which has backup result messages in nodeA. output.txt is saved at the directory on which a user ran "cubrid backupdb" command.

     [nodeA]$ vi $CUBRID_DATABASES/testdb/log/output.txt
     
     [ Database(testdb) Full Backup start ]
     
     - num-threads: 2
     
     - compression method: NONE
     
     - sleep 10 millisecond per 1M read.
     
     - backup start time: Fri Mar 20 18:18:53 2015
     
     - number of permanent volumes: 2
     
     - HA apply info: testdb 1426495463 12922 16192
     
     - backup progress status
     
     ...
     

     Make the below script and run. At "HA apply info" on the above output, put the first number, 1426495463 into $db_creation, the second number, 12922 into $pageid, and the third number, 16192 into $offset; put testdb into db_name, the database name and nodeA into master_host, the master node's host name.

     [nodeB]$ vi update.sh
     
     #!/bin/sh
     
     db_creation=1426495463
     page_id=12922
     offset=16192
     db_name=testdb
     master_host=nodeA
     
     repl_log_home_abs=$CUBRID_DATABASES
     
     repl_log_path=$repl_log_home_abs/${db_name}_${master_host}
     
     local_db_creation=`awk 'BEGIN { print strftime("%m/%d/%Y %H:%M:%S", $db_creation) }'`
         csql_cmd="\
         INSERT INTO \
                 db_ha_apply_info \
         VALUES \
         ( \
                 '$db_name', \
                 datetime '$local_db_creation', \
                 '$repl_log_path', \
                 $page_id, $offset, \
                 $page_id, $offset, \
                 $page_id, $offset, \
                 $page_id, $offset, \
                 $page_id, $offset, \
                 $page_id, $offset, \
                 NULL, \
                 NULL, \
                 NULL, \
                 0, \
                 0, \
                 0, \
                 0, \
                 0, \
                 0, \
                 0, \
                 NULL \
         )"
     
     # Insert nodeA's HA info.
     csql --sysadm -u dba -c "$csql_cmd" -S testdb
     

     [nodeB]$ sh update.sh
     

     Check if the data is successfully inserted.

     [nodeB]$ csql -u dba -S testdb
     
     csql> ;line on
     csql> SELECT * FROM db_ha_apply_info;
     

3. Copy nodeA's active log into nodeB

   In nodeA's active log, there exists a number information of the archive log which is recently created. This information is needed for archive logs, which has been created after the time in which the active log had been copied, to be automatically copied by copylogdb process after the start of HA service of nodeC; therefore, the user just needs to manually copy archive logs which has been created before copying an active log.

   • Start cub_master which manages HA connection in nodeB.

     [nodeB]$ cub_master
     

   • Start copylogdb process for nodeA on nodeB.

     [nodeB]$ cubrid heartbeat copylogdb start testdb nodeA
     

   • Check if an active log of nodeA is copied.

     To copy the current active log means to obtain the number information of the last archive log. Archive logs after this time are automatically copied after the process [5. Start HA server in nodeB].

     [nodeB]$ cd $CUBRID_DATABASES/testdb_nodeA
     [nodeB]$ ls
     testdb_lgar_t  testdb_lgat  testdb_lgat__lock
     

   • Stop copylogdb process for nodeA on nodeB.

     [nodeB]$ cubrid heartbeat copylogdb stop testdb nodeA
     

   • Stopo nodeB's cub_master.

     [nodeB]$ cubrid service stop
     

4. Copy required log files after restoring a database into nodeB

   • Copy all archive logs of nodeA into nodeB.

     [nodeB]$ cd $CUBRID_DATABASES/testdb_nodeA
     [nodeB]$ scp -l 131072 cubrid@nodeA:$CUBRID_DATABASES/testdb/log/testdb_lgar0* .
     

     In this process, please note that all replicated archive logs which are required for replication should exist.

     Note

     scp command's -l option is an option to control the amount of copying file; when you copy files between nodes, use this properly as considering I/O overhead. The unit is Kbits. 131072 is 16MB.

     Note

     Copy files of which the names end with a number. Here, we copied with "testdb_lgar0*" because numbers of names of all replication log files are started with 0.

     Note

     When required logs are already deleted

     Archive logs which are created between backup time and active log creation time can be deleted if the value of log_max_archives is not large enough; in this case, an error can occur when running the below process, [5. Start HA service on nodeB ...].

     When an error occurs, enlarge the value of log_max_archives enough and proceed again backup-and-restore from the beginning.

5. Start HA service on nodeB; restart copylogdb and applylogdb processes on nodeA and nodeC

   [nodeB]$ cubrid heartbeat start
   

   [nodeA]$ cubrid heartbeat applylogdb start testdb nodeB
   [nodeA]$ cubrid heartbeat copylogdb start testdb nodeB
   
   [nodeC]$ cubrid heartbeat applylogdb start testdb nodeB
   [nodeC]$ cubrid heartbeat copylogdb start testdb nodeB
   

How to Detect Replication Mismatch¶

Replication mismatch between replication nodes, indicating that data of the master node and the slave node is not identical, can be detected to some degree by the following process. You can also use checksumdb utility to detect a replication inconsistency. However, please note that there is no more accurate way to detect a replication mismatch than by directly comparing the data of the master node to the data of the slave node. If it is determined that there has been a replication mismatch, you should rebuild the database of the master node to the slave node (see Rebuilding Replication Script.)

• Execute cubrid statdump command and check Time_ha_replication_delay. When this value is bigger, replication latency can be larger; the bigger latency time shows the possibility of the larger replication mismatch.

• On the slave node, execute cubrid applyinfo to check the "Fail count" value. If the "Fail count" is 0, it can be determined that no transaction has failed in replication (see applyinfo.)

  [nodeB]$ cubrid applyinfo -L /home/cubrid/DB/testdb_nodeA -r nodeA -a testdb
  
   *** Applied Info. ***
  Committed page                 : 1913 | 2904
  Insert count                   : 645
  Update count                   : 0
  Delete count                   : 0
  Schema count                   : 60
  Commit count                   : 15
  Fail count                     : 0
  ...
  

• To check whether copying replication logs has been delayed or not on the slave node, execute cubrid applyinfo and compare the "Append LSA" value of "Copied Active Info." to the "Append LSA" value of "Active Info.". If there is a big difference between the two values, it means that delay has occurred while copying the replication logs to the slave node (see applyinfo.)

  [nodeB]$ cubrid applyinfo -L /home/cubrid/DB/testdb_nodeA -r nodeA -a testdb
  
  ...
  
   *** Copied Active Info. ***
  DB name                        : testdb
  DB creation time               : 11:28:00.000 AM 12/17/2010  (1292552880)
  EOF LSA                        : 1913 | 2976
  Append LSA                     : 1913 | 2976
  HA server state                : active
  
   ***  Active Info. ***
  DB name                        : testdb
  DB creation time               : 11:28:00.000 AM 12/17/2010  (1292552880)
  EOF LSA                        : 1913 | 2976
  Append LSA                     : 1913 | 2976
  HA server state                : active
  

• If a delay seems to occur when copying the replication logs, check whether the network line speed is slow, whether there is sufficient free disk space, disk I/O is normal, etc.

• To check the delay in applying the replication log in the slave node, execute cubrid applyinfo and compare the "Committed page" value of "Applied Info." to the "EOF LSA" value of "Copied Active Info.". If there is a big difference between the two values, it means that a delay has occurred while applying the replication logs to the slave database (see applyinfo.)

  [nodeB]$ cubrid applyinfo -L /home/cubrid/DB/testdb_nodeA -r nodeA -a testdb
  
   *** Applied Info. ***
  Committed page                 : 1913 | 2904
  Insert count                   : 645
  Update count                   : 0
  Delete count                   : 0
  Schema count                   : 60
  Commit count                   : 15
  Fail count                     : 0
  
   *** Copied Active Info. ***
  DB name                        : testdb
  DB creation time               : 11:28:00.000 AM 12/17/2010  (1292552880)
  EOF LSA                        : 1913 | 2976
  Append LSA                     : 1913 | 2976
  HA server state                : active
  ...
  

• If the delay in applying the replication logs is too long, it may be due to a transaction with a long execution time. If the transaction is performed normally, a delay in applying the replication logs may normally occur. To determine whether it is normal or abnormal, continuously execute cubrid applyinfo and check whether applylogdb continuously applies replication logs to the slave node or not.

• Check the error log message created by the copylogdb process and the applylogdb process (see the error message).

• Compare the number of records on the master database table to that on the slave database table.

checksumdb¶

checksumdb provides a simple way to check replication integrity. Basically, it divides each table from a master node into fixed-size chunks and then calculates CRC32 values. The calculation itself, not the calculated value, is then replicated through CUBRID HA. Consequently, by comparing CRC32 values calculated on master and slave nodes, checksumdb can report the replication integrity. Note that checksumdb might affect master's performance even though it is designed to minimize the performance degradation.

cubrid checksumdb [options] <database-name>@<hostname>

• <hostname> : When you initiates checksum calculation, you need to specify the hostname of a master node. When you need to get the result after the calculation is completed, specify the hostname of a node you want to check.

-c, --chunk-size=NUMBER¶

You can specify the number of rows to select for each CRC32 calculation. (default: 500 rows, minimum: 100 rows)

-s, --sleep=NUMBER¶

checksumdb sleeps the specified amount of time after calculating each chunk (default: 100 ms)

-i, --include-class-file=FILE¶

You can specify tables to check the replication mismatch by specifying the -i FILE option. If it is not specified, entire tables will be checked. Empty string, tab, carriage return and comma are separators among table names in the file.

-e, --exclude-class-file=FILE¶

You can specify tables to exclude from checking the replication mismatch by specifying the -e FILE option. Note that either -i or -e can be used, not both.

-t, --timeout=NUMBER¶

You can specify a calculation timeout with this option. (default: 1000 ms) If the timeout is reached, the calculation will be cancelled and will be resumed after a short period of time.

-n, --table-name=STRING¶

You can specify a table name to save checksum results. (default: db_ha_checksum)

-r, --report-only¶

After checksum calculation is completed, you can get a report with this option.

--resume¶

When checksum calculation is aborted, you can resume the calculation using this option.

--schema-only¶

When this option is given, checksumdb does not calculate CRC32 but only check schema of each table

--cont-on-error¶

Without this option, checksumdb halts on errors.

The following example shows how to start checksumdb

cubrid checksumdb -c 100 -s 10 testdb@master

When no replication mismatch found,

$ cubrid checksumdb -r testdb@slave
================================================================
 target DB: testdb@slave (state: standby)
 report time: 2016-01-14 16:33:30
 checksum table name: db_ha_checksum, db_ha_checksum_schema
================================================================

------------------------
 different table schema
------------------------
NONE

----------------------------------------------------------------
table name  diff chunk id   chunk lower bound
----------------------------------------------------------------
NONE

--------------------------------------------------------------------------------------
table name  total # of chunks       # of diff chunks        total/avg/min/max time
--------------------------------------------------------------------------------------
t1              7                       0                       88 / 12 / 5 / 14 (ms)
t2              7                       0                       96 / 13 / 11 / 15 (ms)

When there is a replication mismatch in table t1,

$ cubrid checksumdb -r testdb@slave
================================================================
 target DB: testdb@slave (state: standby)
 report time: 2016-01-14 16:35:57
 checksum table name: db_ha_checksum, db_ha_checksum_schema
================================================================

------------------------
 different table schema
------------------------
NONE

----------------------------------------------------------------
table name  diff chunk id   chunk lower bound
----------------------------------------------------------------
t1              0               (id>=1)
t1              1               (id>=100)
t1              4               (id>=397)

--------------------------------------------------------------------------------------
table name  total # of chunks       # of diff chunks        total/avg/min/max time
--------------------------------------------------------------------------------------
t1              7                       3                       86 / 12 / 5 / 14 (ms)
t2              7                       0                       93 / 13 / 11 / 15 (ms)

When there is a schema mismatch in table t1,

$ cubrid checksumdb -r testdb@slave
================================================================
 target DB: testdb@slave (state: standby)
 report time: 2016-01-14 16:40:56
 checksum table name: db_ha_checksum, db_ha_checksum_schema
================================================================

------------------------
 different table schema
------------------------
<table name>
t1
<current schema - collected at 04:40:53.947 PM 01/14/2016>
CREATE TABLE [t1] ([id] INTEGER NOT NULL, [col1] CHARACTER VARYING(20), [col2] INTEGER, [col3] DATETIME, [col4] INTEGER,  CONSTRAINT [pk_t1_id] PRIMARY KEY  ([id])) COLLATE iso88591_bin
<schema from master>
CREATE TABLE [t1] ([id] INTEGER NOT NULL, [col1] CHARACTER VARYING(20), [col2] INTEGER, [col3] DATETIME,  CONSTRAINT [pk_t1_id] PRIMARY KEY  ([id])) COLLATE iso88591_bin

* Due to schema inconsistency, the checksum difference of the above table(s) may not be reported.
----------------------------------------------------------------
table name  diff chunk id   chunk lower bound
----------------------------------------------------------------
NONE

--------------------------------------------------------------------------------------
table name  total # of chunks       # of diff chunks        total/avg/min/max time
--------------------------------------------------------------------------------------
t1              7                       0                       95 / 13 / 11 / 16 (ms)
t2              7                       0                       94 / 13 / 11 / 15 (ms)

HA Error Messages¶

The following are error messages about the errors which can be reasons for the replication mismatch.

ha_make_slavedb.sh Script¶

To rebuild replications, use the ha_make_slavedb.sh script. This script is located in $CUBRID/share/scripts/ha. Before rebuilding replications, the following items must be configured for the environment of the user. This script is supported since the version 2008 R2.2 Patch 9 and its configuration is different from 2008 R4.1 Patch 2 or earlier. This document describes it in CUBRID 2008 R4.1 Patch 2 or later.

• target_host : The host name of the source node (master node in general) for rebuilding replication. It should be registered in /etc/hosts. A slave node can rebuild replication by using the master node or the replica node as the source. A replica node can rebuild replication by using the slave node or another replica node as the source.
• repl_log_home : Specifies the home directory of the replication log of the master node. It is usually the same as $CUBRID_DATABASES. You must enter an absolute path and should not use a symbolic link. You also cannot use a slash (/) after the path.

The following are optional items:

• db_name : Specifies the name of the database to be replicated. If not specified, the first name that appears in ha_db_list in $CUBRID/conf/cubrid_ha.conf is used.
• backup_dest_path : Specifies the path in which the backup volume is created when executing backupdb in source node for rebuilding replication.
• backup_option : Specifies necessary options when executing backupdb in the source node in which replication will be rebuilt.
• restore_option : Specifies necessary options when executing restoredb in the target node in which replication will be rebuilt.
• scp_option : Specifies the scp option which enables backup of source node in which replication is rebuilt to copy into the target node. The default option is -l 131072, which does not impose an overload on network (limits the transfer rate to 16 MB).

Once the script has been configured, execute the ha_make_slavedb.sh script in the target node in which replication will be rebuilt. When the script is executed, rebuilding replication happens in a number of phases. To move to the next stage, the user must enter an appropriate value. The following are the descriptions of available values.

• yes : Keeps going.
• no : Does not move forward with any stages from now on.
• skip: Skips to the next stage. This input value is used to ignore a stage that has not necessarily been executed when retrying the script after it has failed.